AO  L ADA052622 


AFHRL-TR-77-72 


\ 


PROCEEDINGS  OF  THE  SIMULATOR  EFFECTIVENESS 
RESEARCH  PLANNING  MEETING 


Edited  by 

Edward  E.  Eddowet 


FLYING  TRAINING  DIVISION 
Williams  Air  Force  Base,  Arizona  85224 


D D C 

r.. 

^ APR  14  »78  : 

nl 


AIR  FORCE  SYSTEMS  COMMAND 

BROOKS  AIR  FORCE  BASE, TEXAS  78235 


J 


NOTICE 


When  U.S.  Government  drawings,  specifications,  or  other  data  are  used 
for  any  purpose  other  than  a definitely  related  Government 
procurement  operation,  the  Government  thereby  incurs  no 
responsibility  nor  any  obligation  whatsoever,  and  the  fact  that  the 
Government  may  have  formulated,  furnished,  or  in  any  way  supplied 
the  said  drawings,  specifications,  or  other  data  is  not  to  be  regarded  by 
implication  or  otherwise,  as  in  any  manner  licensing  the  holder  or  any 
other  person  or  corporation,  or  conveying  any  rights  or  permission  to 
manufacture,  use,  or  sell  any  patented  invention  that  may  in  any  way 
be  related  thereto. 

This  final  report  was  submitted  by  Rying  Training  Division,  Air  Force 
Human  Resources  Laboratory,  Williams  Air  Force  Base,  Arizona  85224, 
under  project  1 123,  with  HQ  Air  Force  Human  Resources  Laboratory 
(AFSC),  Brooks  Air  Force  Base,  Texas  78235. 

This  report  has  been  reviewed  and  cleared  for  open  publication  and/or 
public  release  by  the  appropriate  Office  of  Information  (OO  in 
accordance  with  AFR  190-17  and  DoDD  5230.9.  There  is  no  objection 
to  unlimited  distribution  of  this  report  to  the  public  at  large,  or  by 
DDC  to  the  National  Technical  Information  Service  (NTIS). 

This  technical  report  has  been  reviewed  and  is  approved  for  publication. 

EDWARD  E.  EDDOWES,  Technical  Advisor 
Rying  Training  Division 


DAN  D.  FULGHAM,  Colonel,  USAF 
Commander 


FHRL-TR-77-72 


security  classification  of  this  page  rmian  Dmim  EnlaracV 


REPORT  DOCUMENTATION  PAGE 


READ  INSTRUCTIONS 
BEFORE  COMPLETING  FORM 


2.  GOVT  ACCESSION  NO. 


^OCEEDINGS  OF  THE  SIMULATOR  ^FECTIVENESS 
iffiSE  ARCH  PLANNING  Feting. 


Proceedings 
21  -22  April  1976 


S PERFORMING  organization  NAME  AND  AOORESS 

Rying  Training  Division 

Air  Force  Human  Resources  Laboratory 

WQIiams  Air  Force  Base,  Arizona  85224 


e.  PERFORMING  ORG.  REPORT  NUMBER 


8.  CONTRACT  OR  GRANT  NUMBERr>; 


10.  PROGRAM  ELEMENT.  PROJECT,  TASK 
WORK  UNIT  NUMBERS 

112303|28 
11230^1 


II.  CONTROLLING  OFFICE  NAME  AND  ADDRESS 

HQ  Air  Force  Human  Resources  Laboratory  (AFSC) 
Brooks  Air  Force  Base,  Texas  78235 


4 MONITORING  AGENCY  NAME  8 ADORESSr/f  dlf/eranf  from  Controlling  Ollier)  IS.  SECURITY  I 


.^///£3J  jyiMJ 


<6.  OISTmeuTlOM  STATeMENT  (of  fhU  Report) 

Approved  for  public  release; distribution  unlimited. 


17.  distribution  statement  (of  tho  obatrmet  onterod  In  Block  20,  If  dlltoront  from  Report) 


Unclassified 


15e.  declassification  DOWNGRADING 
SCHEDULE 


[18.  SUPPLEMENTARY  NOTES 


[19.  key  words  (Continue  on  revere#  side  // neceeeery  mnd  Identity  by  block  number) 


flight  simulators  training  devices 

flight  training  training  effectiveness 

simulator  effectiveness  transfer  of  training 

simulator  motion  - . visual  simulation 

'Z-  .-v  ■_  ~ ' 

so  abstract  fCenrtnue  ort  revere#  aide  If  neceaamry  mnd  Identity  by  block  number) 

- The  proceedings  of  a Simulator  Effectiveness  Research  PlannifTg  Meetin^eld  at  the  Flying  Training  Division, 

Air  Force  Human  Resources  Laboratory ^^Williams  Air  Force  Base,  Arizona  on  21—22  April  1976  are  reported.  The 
objective  of  the  nieetineeas  to  identify  simulator  training  requirements  and  to  determine  the  availabdity  of  research 
facilities  and  personnel  to  use  in  studies  of  the  training  effectiveness  of  simulator  motion  and  visual  systems.  The 
meeting  consisted  of  two  sessions.  During  the  first  session  attendees  were  divided  into  four  groups  which  studied:  (a) 
capabilities  and  characteristics  of  simulators?  (b)  simulator  requirements  for  flying  training,*  (c)  measurement  of 
simulator  training  effectiveness,'  and  (d)  management  and  utilization  of  simulators.  During  the  second  session  the 
findings  of  the  study  groups  were  summarized  and  discussed  in  a general  meeting  of  all  attendees. 

OD  I JAN  73  1473  eoiTioN  OF  I NOV  8S  IS  OBSOLETE  Unclassified 

SECURITY  CLASSIFICATION  OF  THIS  PAGE  Dere  Entered) 


SECURITY  CLASSIFICATION  OP  THIS  PAGE^HTimi  D»tm  Bnt0Tmd) 


SECURITY  CL ASSiriCATlOH  OP  THIS  PAGEfWh^n  0«f»  Enf»r*d> 


J 


TABLE  OF  CONTENTS 

Page 


I.  Introduction  3 

II.  Report  of  Group  Discussion  3 


Study  Group  I Discussion  TOPIC:  Capabilities  and  Characteristics  of  Simulators  . . 
Study  Group  2 Discussion  TOPIC:  Simulator  Requirements  for  Hying  Training  . . 
Study  Group  3 Discussion  TOPIC:  Measurement  of  Simulator  Training  Effectiveness 
Study  Group  4 Discussion  TOPIC:  Management  of  Procurement  and  Utilization 


of  Simulators  6 

Recap  of  Discussion 9 

Appendix  A:  List  of  Attendees  11 

Appendix  B:  Agenda  of  Simulator  Training  Research  Planning  Meeting 13 


Appendix  B:  Agenda  of  Simulator  Training  Research  Planning  Meeting 13 


LA  ^ U) 


PROCEEDINGS  OF  THE  SIMULATOR  EFFECTIVENESS 
RESEARCH  PLANNING  MEETING 


I.  INTRODUCTION 

The  Flying  Training  Division  of  the  Air  Force 
Human  Resources  Laboratory  (AFHRL/FT)  was 
tasked  to  develop  and  conduct  a research  program 
designed  to  generate  a description  of  the  training 
effectiveness  of  flight  simulator  motion  and  visual 
systems.  A working  meeting  was  held  at  Williams 
Air  Force  Base,  Arizona,  on  the  21st  and  22nd of 
April,  1976  with  the  objective  of  identifying  simu- 
lator training  requirements  and  determining  the 
availability  of  research  facilities  and  personnel  for 
use  in  studies  of  the  training  effectiveness  of  simu- 
lator motion  and  visual  systems.  This  report 
documents  the  proceedings  of  the  meeting.  A list 
of  attendees  is  presented  in  Appendix  A. 

Colonel  J.  D.  Boren,  Chief,  AFHRL/FT, 
opened  the  meeting  and  welcomed  the  participants 
to  the  research  planning  sessions.  Dr.  E.  E. 
Eddowes,  AFHRL/FT,  completed  the  opening 
session  with  a review  of  the  organization  of  the 
two-day  meeting.  He  underscored  the  meeting’s 
objective  and  emphasized  the  opportunity  the 
meeting  provided  to  assist  in  the  development  of 
the  research  program  needed  to  achieve  the 
objective. 

Following  the  opening  session  of  the  meeting, 
the  participants  divided  into  four  groups  to  discuss 
various  aspects  of  simulator  training  effectiveness 
research  planning.  The  results  of  these  discussions 
are  reported  in  the  following  section  of  this  report. 
The  agenda  is  presented  in  Appendix  B. 


II.  REPORT  OF  GROUP  DISCUSSIONS 


Study  Group  1 Discussion 
TOPIC:  Capabilities  and 


The  group  expressed  diverse  points  of  view 
about  many  facets  of  the  research  discussion 
questions.  Subsequent  discussion  focused  on  an  at- 
tempt to  better  define  an  approach  and  agree  on 
an  objective  for  the  discussion  that  was 
manageable. 

The  objectives  selected  were:  (a)  Identify  avail- 
able devices  which  should  be  considered  for  use  in 
a future  experimental  program,  and  (b)  identify 
simulator  motion  and  visual  system  characteristics 
or  independent  variables  that  must  be  considered 
in  any  future  experimental  program. 


Simulator  Categories:  The  first  step  toward  the 
initial  objective  was  to  categorize  simulators 
according  to  the  mission  characteristics  of  the  air- 
craft simulated.  Two  categories  were  selected: 
Cargo/bomber  and  fighter/trainer.  Since  the  orig- 
inal issue  to  be  addressed  was  motion,  later 
changed  to  motion  and  vision  since  the  two  are 
virtually  inseparable,  the  group  looked  first  at 
simulators  with  appropriate  motion  systems.  A 
need  was  recognized  to  ascertain  the  general  char- 
acteristics of  the  motion  system  and  other  infor- 
mation such  as  availability,  research  support 
capability,  and  the  type  of  visual  system. 

Eleven  simulators  or  sets  of  simulators  were 
identified  for  use  on  cargo/bomber  motion/vision 
research  as  follows:  (a)  Facilities  of  two  airlines 
and  an  aircraft  manufacturer’s  training  simulators, 
(b)  Four  engineering  simulators,  (c)  Simulator 
manufacturing  developmental  system,  and  (d) 
Three  operational  training  simulators.  Eight  simu- 
lators were  identified  for  use  on  fighter/trainer 
motion/vision  research  as  follows:  (a)  Four  engi- 
neering simulators,  (b)  One  training  research 
simulator;  and  (c)  Three  operational  simulators.  (It 
should  be  noted  that  all  but  one  of  the  simulators 
identified  are  equipped  with  a visual  system.) 

Selection  criteria  were  established  which  led  to 
consideration  of  the  higher  capability  systems  (5- 
or  6-degree-of-freedom  motion  rather  than  2-  or 
3-degree-of-freedom  motion  systems).  Thus,  for 
near-term  experimental  efforts,  the  most  capable 
simulators  available  should  be  used. 

Motion  Cueing  Systems:  The  next  step  was  to 
consider  other  motion  simulation  devices  (G-seats, 
G-suits,  buffet,  display  timing  and  sound).  An 
attempt  was  made  to  identify  other  systems  with 
these  cue-generation  capabilities. 

The  following  devices  other  than  training  or 
engineering  simulators  were  considered  (lab-type 
systems):  (a)  Aerospace  Medical  Research  Labora- 
tory (AMRL)  System  (rotational  arc  tracking 
system)  which  includes  motion/visual  delays  and 
various  rotation  drive  algorithms,  and  (b) 
Grumman  Lab  System  which  includes  the  capa- 
bilities for  studying  acceleration  thresholds  and 
washout  profiles  or  signatures.  There  were  also 
discussions  of  study  efforts  such  as  are  being 
accomplished  at  Massachusetts  Institute  of  Tech- 
nology (MIT)  directed  toward  modeling  human 
motion  sensory  mechanisms  including  the 
vestibular,  proprioceptive,  tactile  and  visually 
induced  motion  cues. 
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Summary  of  Group  I Discussion:  There  is  some 
information  already  (a)  available  on  motion/visual 
system  effectiveness.  A first  step  should  be  to 
identify  what  is  already  known  before  another 
experimental  effort  is  undertaken.(b)  The  impres- 
sion that  the  experimental  approach  is  not  the 
final  answer  to  the  research  questions  was  ex- 
pressed, (c)  There  are  a body  of  knowledge  and  a 
number  of  simulator  motion  experts  available  to 
provide  information  to  be  applied  in  developing 
short-term  answers  on  the  effectiveness  of  simu- 
lator motion  systems. (d)  Analysis  of  the  cues  dis- 
played used  by  pilots  in  simulators  may  provide 
the  key  to  being  able  to  extrapolate  forward  to 
define  future  simulator  requirements.  Validation 
could  then  be  provided  through  experimental 
evaluation  with  the  experiments  using  existing 
devices,  (e)  Of  the  devices  identified  for  fighter/ 
trainer  motion  research,  concern  was  expressed  for 
ones  which  have  known  or  suspected  deficiencies, 
such  as  excessive  time  delays  in  the  motion  drive 
system.  The  feeling  was  that  these  kinds  of  defi- 
ciencies should  be  corrected  as  soon  as  possible. 
Although  the  Advanced  Simulator  for  Pilot  Train- 
ing (ASPT)  is  considered  adequate  for  research 
involving  T-37  flight  performance  envelopes,  for 
work  on  the  motion  requirement  for  simulators  of 
higher  performance  aircraft  such  as  the  F-16  it 
should  be  improved  particularly  in  the  area  of 
motion  algorithms  and  computation  iteration 
rates.  (0  One  recommendation  was  that  research 
should  be  focused  in  the  visual  area  to  develop 
ways  that  visual  systems  can  substitute  for  motion; 
for  example,  wider  field  of  view  (FOV)  to  foster 
visually  induced  motion.  Consideration  of  the  cost 
aspects  of  visual  systems  led  to  expectations  that 
computer  image  generated  (CIG)  systems  should 
get  cheaper  within  the  next  10  years,  whereas 
motion  platform  system  costs  are  likely  to 
increase,  (g)  The  final  major  point  of  discussion 
focused  on  where  we  are  in  the  development  of 
simulator  motion  systems.  A question  was  raised 
about  the  validity  of  the  results  of  some  of  the 
recent  studies  using  ASPT  which  show  no 
significant  differences  in  transfer  of  training  to  the 
aircraft  for  students  trained  with  or  without 
platform  motion. 

Study  Group  2 Discussion 
TOPIC:  Simulator  Requirements  for 
Flying  Training 

Command  representatives  were  present  from 
Air  Defense  Command  (ADCOM),  Air  Training 
Command  (ATC),  Strategic  Air  Command  (SAC), 
and  Tactical  Air  Command  (TAC).  An  orientation 
briefing  was  given  which  presented  the  objectives 


of  the  working  group.  In  order  to  provide  an  ade- 
quate definition  of  simulator  subsystem  require- 
ments, each  command  representative  was 
requested  to  address  the  following  questions:  (a) 
Type  of  aircraft  (fighter,  trainer,  bomber,  trans- 
port, etc.);  (b)  Key  missions  (air-to-air,  air-to- 
surface,  instruments,  navigation,  etc.);  (c) 
Projected  simulator  uses  (initial  training,  profi- 
ciency maintenance,  simulator  flight-checks,  etc.); 
(d)  Experience  level  (novice  pilots,  UPT  graduates, 
high-time,  etc.);  (e)  Projected  needs  in  the  areas  of 
visual  and  motion  simulation;  (f)  Data  required  to 
influence  decisions  on  the  procurement  of  simula- 
tor subsystems-especially  motion  (e.g.,  transfer  of 
training  from  simulator  to  aircraft,  performance  in 
the  simulator,  pilot  acceptance,  motion  discrimina- 
tion, etc.);  and  (g)  Priorities  (definition  of  critical 
task  areas;  selection  of  data  points  to  maximize 
information,  etc.).  The  planning  effort  to  deter- 
mine the  incremental  training  effectiveness  of 
platform  motion  was  used  as  an  example. 
Following  the  introductory  session,  the  simulator 
requirements  for  ADCOM,  ATC,  SAC,  and  TAC 
were  identified  and  discussed. 

Summary  of  Group  2 Discussion:  The  following 
points  of  agreement  and  areas  of  concern  seemed 
to  emerge  during  the  discussion:  (a)  Data  are 
required  in  order  to  assess  the  effectiveness  of  plat- 
form motion.  It  was  agreed  that  transfer  of  train- 
ing data  are  required,  (b)  It  was  agreed  that  the 
maneuvers  selected  for  motion  research  should 
exercise  the  motion  system  to  as  great  an  extent  as 
possible  (Because  of  their  relatively  higher  motion 
cue  requirements,  aerobatic,  or  combat-type 
maneuvers,  were  suggested  as  more  suitable  for 
research  on  the  training  effectiveness  of  simulator 
motion  than  basic  aircraft  control  training 
maneuvers.),  (c)  Concern  was  expressed  over  the 
need  for  motion  cues  for  training  emergency  re- 
actions in  which  there  exists  no  safe  way  of 
assessing  transfer  to  the  aircraft,  (d)  Controversy 
was  apparent  in  the  extent  to  which  simulator  cues 
are  necessary  for  experienced  as  opposed  to  novice 
pilots,  (e)  It  was  agreed  that  a need  exists  to  assess 
the  saliency  of  motion  cues  for  various  flight 
maneuvers-both  subjective  and  objective  methods 
were  suggested,  and  (0  The  question  of  genera- 
lizing study  results  from  one  motion  system  to 
another,  as  well  as  from  one  aircraft  to  another, 
was  raised  and  is  seen  as  a serious  problem. 
Throughout  the  session,  it  was  readily  apparent 
that  the  role  of  motion  cueing  (in  flight  simulation 
training)  is  largely  unknown  and  should  be 
addressed  in  future  research  programs. 
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Study  Group  3 Discussion 
TOPIC:  Measurement  of  Simulator 
Training  Effectiven^ 


Group  3 addressed  research  methodologies 
which  can  be  employed  in  the  study  of  simulator 
training  effectiveness.  With  the  emphasis  on  the 
contribution  of  motion  platforms  to  training  ef- 
fectiveness, three  types  of  experimental  designs 
were  identified:  motion  discrimination  com- 
parisons, in-simulator  performance  studies,  and 
training  effectiveness  research. 

Simple  Motion  Discrimination  Comparisons. 
Two  types  of  “evaluations”  of  motion  systems 
have  come  from  the  comparison  of  motion  cues 
generated  by  the  simulator.  In  the  first,  the  pilot 
flies  the  simulator  with  the  motion  platform  on 
and  with  it  off,  and  then  is  asked  if  he  perceived 
any  difference  in  the  sensation  of  motion. 
Normally,  the  pilot  is  not  told  beforehand  that  he 
is  going  to  be  asked  about  motion,  since  it  seems 
that  if  he  knows  the  question,  he  can  deliberately 
twil'.h  the  control  stick  and  detect  the  simulator’s 
configuration  from  the  resulting  jarring  (or  lack  of 
it).  Informal  demonstrations  of  this  type  of  com- 
parison on  simulators  with  wide  angle  visual 
systems  are  often  dramatic,  with  the  pilot  con- 
vinced that  the  simulator  was  moving  when  in  fact 
it  was  not. 

The  second  type  of  comparison  involves  the 
experienced  pilot  comparing  the  motion  cueing  of 
the  simulator  with  the  aircraft  motion  he  has 
known  as  a pilot.  The  result  of  the  comparison  is  a 
judgement  about  the  fidelity  of  the  motion  cues. 
Currently,  such  comparisons  are  numerous  as  more 
and  more  representatives  of  operational  commands 
and  high  level  staff  agencies  visit  simulator  facili- 
ties to  learn  more  about  available  simulation  hard- 
ware and  make  decisions  about  upcoming 
procurements. 

In  both  of  these  cases,  the  results  of  the  com- 
parison is  some  statement  about  the  sensation  of 
motion  or  fidelity  of  motion  cues  generated  by  the 
motion  system.  While  it  is  important  to  know 
what  types  of  cues  the  simulator  generates,  just 
how  these  cues  relate  to  training  effectiveness  is 
less  than  clear.  Fidelity  is  not  always  directly  pro- 
portional to  training  effectiveness.  Many  skills  can 
be  trained  and  maintained  at  a desired  level  of 
proficiency  with  a low  fidelity  training  device. 

The  hazard  in  the  simple  motion  comparisons  is 
their  intuitive,  instant  compatibility  with  the 
observer’s  desire  to  form  his  opinions  about 
motion  cueing  systems.  Evaluations  are  formed 


quickly  and  convincingly.  Unfortunately,  such 
evaluations  often  have  little  to  do  with  the  training 
device’s  ability  to  train. 

In-Simulator  Performance  Studies.  In  this  type 
of  design,  the  performance  of  the  pilot  flying  the 
simulator  is  used  as  the  indicator  of  the  effective- 
ness of  a training  phase  using  a given  simulator 
configuration.  Performance  in  the  aircraft  is  not 
observed,  thus  transfer  of  training  data  are  not 
generated.  The  assumption  is  that  performance  in 
the  simulator  approximates  the  performance 
which  would  be  observed  if  the  pilot  were  going  to 
fly  the  aircraft  and  therefore  is  acceptable  for  use 
in  making  transfer  of  training  comparisons.  This 
type  of  design  is  suited  to  flying  tasks  which  can- 
not be  flown  in  the  aircraft  during  training  (certain 
critical  emergencies,  for  example),  and  there  are 
data  which  indicate  that  a pilot’s  simulator  per- 
formance is  significantly  and  highly  related  to  his 
subsequent  performance  in  the  aircraft.  However, 
where  transfer  of  training  data  can  be  collected, 
this  design  is  less  powerful  than  a direct  test  of 
training  effectiveness. 

Training  Effectiveness  Designs  (Transfer  of 
Training).  As  the  simulator  is  a training  device  and 
its  ultimate  effectiveness  (and  the  effectiveness  of 
various  components)  is  evaluated  in  the  ability  of 
the  pilot  to  fly  his  airplane  after  simulator  train- 
ing, the  most  powerful  experimental  design  for 
studying  motion  is  the  transfer  of  training  design. 
While  it  has  been  pointed  out  that  traditionally 
this  is  a more  costly  design,  in  some  training 
programs,  where  efficient  performance  measure- 
ment techniques  are  tised,  the  cost  is  justified  by 
the  quality  and  the  applicability  of  the  data 
obtained. 

Given  these  three  types  of  research  approaches 
to  training  effectiveness,  short-  and  long-term 
objectives  for  research  were  identified: 

1.  Short-Term  Objective.  The  short-term  ob- 
jective is  to  create  an  information  base  for  making 
procurement  decisions  in  the  near  future.  Since 
procurement  decisions  must  be  made  regardless  of 
the  information  available,  the  goal  of  the  initial 
motion  research  is  to  supply  useful  information  as 
rapidly  as  possible.  This  approach  impacts  the 
types  and  levels  of  independent  and  dependent 
variables  selected  for  each  study  as  well  as  the 
sample  size.  For  example,  the  motion  variable 
which  has  been  used  by  AFHRL/FT  on  initial 
studies  has  compared  training  under  no  platform 
motion  versus  training  under  six-degree-of-freedom 
synergistic  platform  motion.  If  no  difference  in 
transfer  is  found  under  these  extremes  of  the 


motion  variable,  then  the  experimenter  has  tire 
option  of  refining  the  motion  results;  for  example, 
comparing  no  motion  with  three  degree  and  with 
six  degree  of  freedom  platform  motion,  or  mani- 
pulating completely  different  variables.  In  such 
studies,  the  concept  of  “stacking  the  deck  for 
motion”  prevails;  i.e.,  the  selection  of  variables 
and  tasks  in  such  a way  that  motion  has  the 
greatest  probability  of  improving  training,  so  that 
if  no  differential  effect  of  motion  is  observed,  then 
the  researcher  will  have  a high  level  of  confidence 
that  over  all  tasks  and  variables,  there  is  no  differ- 
ence. This  type  of  research  strategy,  based  on  the 
selection  of  critical  variables  and  tasks,  yields  the 
maximum  amount  of  information  per  unit  of 
effort  and  expense. 

2.  Ixmg-Term  Objectives.  Long-term  research 
will  continue  to  investigate  new  motion  systems, 
recognizing  that,  in  all  probability,  the  “motion 
question”  has  neither  a simple  nor  a single  answer. 
Due  to  expected  state-of-the-art  advances  and 
hardware  and  software  improvements,  engineering 
descriptions  of  experimental  apparatus  (software 
packages  as  well  as  hardware  configurations)  must 
be  documented  to  permit  comparisons  of  study 
results  longitudinally. 

Two  additional  areas  which  were  addressed  by 
Group  3: 

Generalization.  Regarding  the  problem  of  gen- 
eralizing results  across  pieces  of  hardware,  e.g., 
generalizing  the  results  of  a study  conducted  on 
ASPT  to  a newer  about-to-be-procured  simulator. 
Group  3 participants  tended  to  see  generalization 
as  a nonproblem.  Simply  put,  in  the  absence  of 
better  data,  management  decisions  should  be  based 
on  data  that  are  available.  The  argument  that 
results  cannot  be  generalized  because  of  differ- 
ences in  hardware  or  software  is  circular  since  each 
new  device  will  differ  in  some  characteristic  from 
its  predecessor. 

Data.  In  all  studies,  it  is  essential  that  not  only 
the  amount  of  any  differences  between  experi- 
mental groups  be  determined,  but  also  how  long 
such  differences  last  during  subsequent  training  or 
flying.  For  example,  in  a study  assessing  the  con- 
tribution of  a motion  platform  to  undergraduate 
pilot  training  or  transition  training  of  an  expe- 
rienced pilot,  a small  advantage  for  pilots  trained 
with  motion  might  be  seen  on  the  first  aircraft  ride 
after  training,  evidence  in  support  of  motion  pro- 
curement. However,  if  two  rides  later,  no  differ- 
ence can  be  detected  between  pilots  trained 
without  motion  and  those  trained  with  motion, 
the  case  for  motion  procurement  is  considerably 
weakened.  The  type  of  data  generated  by  the 


upcoming  series  of  short-temi  studies  must  be  de- 
signed to  give  the  decision-maker  the  maximum 
number  of  alternatives  in  his  selection  of 
investment  strategies. 

Study  Group  4 Discussion 
TOPIC:  Management  of  Procurement  and 
Utilization  of  Simulators 

The  discussion  of  the  group  which  studied  the 
management  of  procurement  and  utilization  of 
simulators  for  flying  training  focused  on  simulator 
concepts  of  utilization,  configuration,  costs  and 
certification.  While  these  discussion  topics  did  not 
emerge  distinctly,  completely,  or  in  this  sequence, 
they  are  presented  as  though  they  did  to  help 
organize  and  simplify  this  report. 

Concepts  of  Simulator  Utilization.  The  most 
obvious  concept  of  the  use  of  flight  simulators  in 
flying  training  is  as  a substitute  for  training  in  the 
aircraft.  This  concept  leads  directly  to  the  notion 
of  the  simulator’s  being  equivalent  to  the  aircraft. 
However,  simulator  users  inevitably  discover  that 
the  simulator  is  not  equivalent  and  cannot  be 
substituted  for  the  aircraft.  The  result  is  the  fre- 
quently encountered  nonacceptance  of  flight 
simulators  by  pilots  and  the  continuing  efforts  of 
simulator  engineers  to  design  more  realistic 
simulators. 

This  conceptual  state  of  affairs,  if  it  has  been 
portrayed  adequately,  can  be  seen  as  an  impetus 
for  the  development  of  increasingly  more  capable, 
more  complex  and  more  costly  flight  simulators. 

The  group  generated  the  following  observations 
with  respect  to  the  current  state  of  concepts  of 
simulator  utilization: 

1.  Instructors  and  student  pilots  wfll  come  to 
accept  simulators  that  work. 

2.  Instructors  can  “train  around”  typical 
simulator  limitations  by  taking  advantage  of  the 
strengths. 

3.  What  training  managers  do  with  their  simu- 
lators is  more  important  than  any  specific 
properties  of  the  equipment. 

Simulator  Configuration.  A major  problem  in 
specifying  the  requirements  for  the  configuration 
of  any  simulator  is  determining  what  the  device  is 
supposed  to  do,  that  is,  how  and  where  it  fits 
within  a flying  training  program.  Very  often  opera- 
tors generate  requirements  based  on  the  kinds  of 
equipment  currently  available.  Frequently,  the 
requirements  describe  a full-mission,  high-fidelity 
trainer  with  all  the  latest  simulation  features,  such 
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as  six -degree-of  freedom  platform  motion,  G-seat, 
G-suit,  large  field  of  view  visual  display  and  a num- 
ber of  instructional  aids.  Such  requirements  are 
typically  supported  by  the  “conservative”  notion 
that  it  is  better  to  specify  a more  capable  than  a 
less  capable  simulator,  because  the  added 
capability  would  cost  more  and  take  longer  to  ob- 
tain if  it  were  added  later  during  the  procurement 
process. 

In  discussing  this  kind  of  problem  related  to 
determining  simulator  configuration  requirements, 
the  following  observations  were  made: 

1.  Improved  analyses  of  flying  training  re- 
quirements are  needed  to  substantiate  the  design 
of  simulators  so  the  configurations  selected  are 
based  on  the  demonstrated  training  effectiveness 
of  simulator  capabilities  rather  than  the  conser- 
vative approach  noted  above,  or  the  relatively 
vague  appeal  that  maximum  capability  generates 
maximum  realism  which  yields  maximum  pilot 
and  instructor  acceptance. 

2.  Research  designs  which  permit  the  deter- 
mination of  the  incremental  transfer  effectiveness 
of  a particular  simulator  ccHifiguratim  (as  it  night  be  em- 
ployed at  any  point  during  the  training  program 
where  it  will  maximally  facilitate  learning)  should 
be  used  to  determine  the  training  effectiveness  of 
simulator  features  before  they  are  specified  for 
procurement.  An  additional  advantage  of  this 
technique  is  a portrayal  of  the  quantitative  re- 
search results  that  enables  simulator  designers  to 
directly  compare  the  training  effectiveness  of 
alternative  simulator  configurations. 

3.  The  question  of  whether  or  not  higher 
fidelity  simulators  are  more  effective  for  beginning 
students  during  their  initial  training,  for  experi- 
enced pilots  during  transition  or  continuing 
training,  or  for  both  beginning  and  experienced 
pilots  at  all  stages  of  training  should  be  tested  and 
the  results  of  the  tests  applied  in  the  definition  of 
simulator  design  requirements. 

Simtdator  Costs.  There  are  really  only  two 
issues  of  any  substance  that  concerns  the  manage- 
ment of  procurement  and  utilization  of  flight 
simulators  for  flying  training:  their  training  effec- 
tiveness and  their  costs.  In  the  preceding  sections, 
discussions  by  the  group  studying  management 
problems  have  been  stated  in  which  both  simulator 
training  effectiveness  and  costs  were  viewed  as 
parts  of  other  major  considerations.  More  atten- 
tion appeared  to  be  focused  on  areas  other  than 
costs.  This  misrepresents  the  content  of  the 
discussions  reported,  and  more  importantly,  misses 
the  point  of  this  latest  renaissance  of  serious 


attention  to  flight  simulation.  Right  simulators  ate 
an  important  concern  in  the  Air  Force  (and  else- 
where) primarily  because  of  costs.  The  trick  is  to 
avoid  having  to  spend  money,  fuel  and  aircraft 
service  life  to  accomplish  any  flying  training  ob- 
jective that  cannot  be  achieved  cost-effectively, 
using  another  training  medium.  The  flight  simu- 
lator is  the  largest  cost  generator,  aside  from  the 
aircraft,  among  the  training  media  currently  being 
employed  for  flying  training  purposes  and  thus  it 
becomes  the  “hot  item”  it  is  today  among  training 
researchers,  simulator  designers,  military  training 
managers  and  congressional  committees.  The 
discussion  related  to  simulator  cost  touched  on  a 
number  of  issues  of  critical  importance  to  the 
achievement  of  optimally  cost-effective  flying 
training  programs  for  the  military  services. 

The  following  observations  emerged  from  the 
discussion  of  simulator  costs: 

1.  Use  of  simulators  in  flying  training  pro- 
grams typically  focuses  on  either  initial  or  contin- 
uation training.  Cost  savings  are  potentially  greater 
ii  simulators  can  be  used  for  continuation  training, 
because  the  mission  aircraft  simulated  routinely 
cost  more  to  operate  than  typical  training  aircraft, 
and  if  simulators  can  satisfy  the  requirements  for 
continuation  training,  there  is  every  reason  to 
expect  that  they  would  be  suitable  also  for  the 
transition  training  which  prepares  the  pilot  trainee 
for  his  operational  assignment.  On  the  other  hand, 
it  is  often  easier  to  demonstrate  cost  savings  in 
initial  training,  because  at  this  point  the  pilot's 
skill  level  is  relatively  low  and  the  opportunity 
exists  to  generate  gratifying  cost  savings  through 
the  use  of  relatively  inexpensive  training  media 
given  the  fact  that  training  aircraft  are  usually  less 
complex  than  operational  aircraft.  Thus,  both 
initial  and  continuing  training  should  be  attacked 
vigorously  to  identify  how  simulators  can  best  be 
employed  to  minimize  costs  and  maximize 
training. 

2.  In  managing  today’s  aircraft  procurement 
programs,  spending  money  for  flying  training 
appears  to  be  a cost  analysis,  cost  estimation 
proposition.  The  purchase  of  flight  simulators  is 
considered  in  the  context  of  whole  program  costs; 
thus  money  spent  on  simulators  will  not  be  avail- 
able for  the  purchase  of  additional  aircraft.  Since 
most  managers  of  aircraft  procurement  programs 
are  pilots  and  pilots  are  known  to  prefer  flying 
aircraft  to  flying  simulators,  asking  them  to 
approve  a simulator  for  aircraft  trade  requires  sub- 
stantial supporting  documentation.  Such  docu- 
mentation needs  to  be  as  solid  as  possible  and. 
thus,  requires  considerable  improvement. 
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3.  The  procedural  routines  for  estimating 
simulator  costs  are  relatively  well  developed  and 
are  not  kept  secret  from  training  researchers. 
There  is  little  evidence,  however,  that  frontline 
training  researchers  consider  simulator  cost  esti- 
mating techniques  in  the  development  of  their 
experimental  procedures  or  in  the  interpretation 
of  their  study  results.  Thus,  the  simulator  program 
cost  analysts  end  up  using  reasonable,  ball  park- 
type  estimates  and  the  training  researchers  seldom 
contribute  very  much  to  their  refinement.  It 
appears  ihat  the  training  effectiveness  of  flight 
simulators  should  be  evaluated  using  research 
strategies  and  procedures  designed  to  generate  data 
that  can  be  used  at  once  by  cost  analysts,  and  that 
training  researchers  will  need  to  emphasize  cost 
analytical  procedures  in  configuring  their  studies 
with  approximately  the  same  vigor  they  have 
learned  to  apply  in  the  specification  of 
experimental  controls  and  statistical  analyses. 

4.  Generally,  the  management  of  procurement 
of  flight  simulators  is  expressed  in  the  documenta- 
tion of  the  amortization  of  simulator  life  cycle 
costs  in  terms  of  the  flying  training  costs  avoid- 
ance which  results  from  use  of  the  simulator.  This 
leads  to  the  popular  notion  of  the  substitution  of 
simulator  training  for  aircraft  training,  noted 
earlier,  which  in  turn  leads  to  a misleading  over- 
simplification of  what  teaches,  who  learns,  and 
why.  Rather  than  resorting  to  accounting  routines 
that  perpetuate  misconceptions,  it  is  preferable  to 
include  in  life  cycle  cost  estimates  all  the  facts  that 
can  be  marshalled  to  serve  as  the  basis  for  a decision 
on  the  available  alternatives  so  that  the  decision 
made  will  not  turn  sour  in  the  future.  For  ex- 
ample, Operation  and  Maintenance  (O&M)  costs 
should  be  included  even  though  they  are  ordinarily 
substantial  and  thus  increase  overall  costs.  Given 
the  complexity  of  many  of  today’s  simulators, 
failure  to  take  into  account  requirements  for  O&M 
support  can  result  very  quickly  in  unsatisfactory 
availability  and  invalid  simulator  training  if  O&M 
funds  are  not  regularly  provided,  and  the  ef- 
fectiveness of  the  simulation  is  degraded  as  a 
consequence. 

Simulator  Certiflcatum.  The  certification  of 
simulators  was  discussed  in  terms  of  the  current 
Navy  and  airlines  programs.  The  Navy  program 
involves  periodic  evaluation  of  the  engineering  or 
physical  characteristics  of  a simulator.  The  airlines 
simulator  certification  efforts  involve  frequent 
engineering  evaluations  and  extend  to  the  main- 
tenance of  currency  and  standardization  of  the 
supporting  instruction  given  in  all  aspects  of  the 
training  program,  including  the  instruction  and 


instructor  techniques  used  in  the  simulator.  Cur- 
rent schemes  for  certification  of  a simulator’s 
training  effectiveness  thus  have  avoided  so  far  the 
difficulties  associated  with  attempting  to  certify 
that  the  simulator  will  teach.  Whether  or  not  any 
instruction  teaches  is  dependent  in  nearly  every 
case  on  what  and  how  well  a trainee  learns.  If  a 
simulator  can  provide  relevant  practice  opportuni- 
ties, then  it  has  training  potential.  After  all,  if  a 
student  can  practice  a flying  task,  he  can  learn 
how  to  perform  it;  and  if  he  can  learn  a flying 
skill,  he  can  transfer  it  to  the  aircraft  (unless  flying 
skills  operate  according  to  different  laws  than 
other  behavioral  skills).  The  certification  problem 
is  not  really  concerned  with  a simulator’s  training 
effectiveness;  that  is  an  evaluation  issue.  Certifica- 
tion relates  to  the  characteristics  of  a simulator 
that  enable  the  simulator  to  generate  accurately, 
and  on  a regular  basis,  the  training  task  for  which 
it  has  been  designed. 

Discussion  of  simulator  certification  yielded 
these  observations: 

1.  Typically,  aircraft  flight  data  are  not  part  of 
simulator  procurements.  The  group  members  with 
experience  in  these  matters  reported  consistent 
problems  with  evaluating  simulator  handling  quali- 
ties, the  simulator’s  aircraft-like  “control  feel,” 
because  of  the  unavailability  of  flight  data  for  use 
in  determining  this  aspect  of  the  certifiability  of 
the  simulator.  The  need  for  flight  data  appears  to 
be  a self-evident  and  essential  element  in  the  certi- 
fication of  a simulator  within  a flying  training 
program.  Simulator  procurement  plans  should 
include  a requirement  for  this  needed  flight  data. 

2.  Frequent  evaluation  of  simulator  engi- 
neering properties  seems  desirable.  This  will  serve 
to  document  the  simulator’s  operational  char- 
acteristics and  provide  checks  on  such  changes  and 
updating  of  the  simulator  as  are  determined  to  be 
required  during  regular  training  operations. 

3.  While  it  is  not  feasible  to  certify  that  a 
simulator  will  train,  it  is  necessary  to  include  in 
certification  procedures,  coverage  of  the  use  of  the 
simulator,  the  employment  of  its  instructional 
features,  and  instructor  techniques  and  training 
criteria.  In  this  manner,  flying  training  program 
managers  can  insure  that  the  simulator  is  doing  a 
reliable  and  valid  job  on  the  training  objectives  it 
supports. 

4.  Given  that  the  certification  of  fliglit  simu- 
lators is  inevitable,  and  the  certification  programs 
can  be  made  to  work,  the  resulting  flying  training 
use  of  certified  simulators  should  lead  to  some 
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form  of  credit  for  simulator  training  time  in  the 
pilot’s  log  of  flight  experience.  That  is,  pilots 
should  be  able  to  count  simulator  time  along  with 
aircraft  flight  time  as  a measure  of  their 
qualifications. 

Recap  of  Discussion 

This  discussion  concemii.g  the  management  of 
procurement  and  utilization  of  simulators  for 
flying  training  has  been  organized  into  four 
sections  dealing  in  turn  with;  concepts  of  utiliza- 
tion, configuration,  costs,  and  certification. 
Regarding  concepts  of  utilization,  it  was  found 
that  instructor  and  trainee  acceptance  was  related 
directly  to  whether  or  not  the  simulator  worked, 
that  instructors  typically  “trained  around”  simula- 
tor deficiencies  to  take  advantage  of  their 
strengths,  and  that  the  simulators’  uses  were  more 
important  than  their  specific  hardware  properties. 
Improved  specification  of  the  training  require- 
ments, use  of  more  powerful  research  procedures 
such  as  the  incremental  transfer  effectiveness 
design  in  evaluating  equipment  alternatives,  and  a 
study  of  the  fidelity,  or  realism,  requirements  for 
effective  employment  of  simulators  in  the  training 
of  pilots  with  different  levels  of  experience  were 
discussed  with  respect  to  simulator  configuration. 
On  the  issue  of  costs,  the  discussion  indicated:  (a) 
that  both  initial  and  continuation  training  were 


valid  targets  for  cost  avoidance  througli  use  of 
simulators,  (b)  that  the  program  costs  of  simu- 
lators are  often  expressed  in  terms  of  aircraft  not 
purchased,  which  results  in  close  scrutiny  being 
given  documentation  supporting  the  use  of  simu- 
lators, (c)  that  training  researchers  should  consider 
the  cost  analyst’s  need  for  training  effectiveness 
data  in  all  phases  of  research  so  the  results  will  in 
fact  be  useful  in  refining  program  cost  estimates, 
and  (d)  that  all  factors  known  to  bear  on  the  life 
cycle  costs  need  be  considered  in  determining 
simulator  cost  amortization  for  use  in  deciding 
among  hardware  alternatives.  Certification  of 
simulators  was  discussed  in  terms  of  a continuing 
need  for  flight  data  for  application  in  evaluating 
and  documenting  the  physical  or  engineering  pro- 
perties of  simulators,  in  terms  of  frequent 
checking  and  update  of  simulator  configuration  so 
records  of  the  simulator  characteristics  will 
support  its  use  in  flying  training,  in  terms  of 
including  all  aspects  of  the  training  program  in  the 
certification  procedures,  not  just  the  simulator 
hardware,  and  finally  in  terms  of  a need  for 
counting  simulator  training  hours  with  aircraft 
training  hours  for  flight  log  purposes  as  an 
indication  of  pilot  experience. 
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APPENDIX  A:  LIST  OF  ATTENDEES 


Grade/Name  Organization 


Capt  Allan  0.  Pelcak 

ATC/XPQC 

Larry  Pohlmann 

AFHRL/FTS 

Capt  D.  P.  Thomas 

CNO 

Norman  W.  King 

AFHRl^FTT 

Edward  E.  Eddowes 

AFHRL/FTT 

Larry  Fogel 

Decision  Sciences,  Inc. 

D.  G.  O’Connor 

Singer- Link 

G.  Kron 

Singer-Link 

R.  F.  New 

Singer- Link 

Cdr  Max  Quitiquit 

CNET  Uaison,  AFHRL/FT 

Harry  Beardsley 

GE 

Col  J.  D.  Boren 

AFHRL/FT 

Thomas  A.  Payne 

Decision  Sciences 

Robert  C.  Houston 

American  Airlines 

James  F.  Smith 

AFHRL/FTS 

Lt  Col  John  F.  Ahlborn 

HQAFSC/DLS 

John  C.  Dusterberry 

NASA/ Ames  Research  Center 

Andrew  Junker 

AMRL/EMT 

Charles  Hopkins 

Aviation  Research  Lab, 
University  of  Illinois 

J . Hensley 

Northrop  Corp 

A.  D.  Windsor 

CNATRA,  Corpus  Christi,  TX 

M.  Freitag 

CNETS 

Gary  Morton 

CNETS 

Jesse  Orlansky 

IDA 

Robert  K.  Jellison 

Decision  Sciences,  Inc. 

Col  Calvin  H.  Markwood 

ASD/SD-24 

Milton  Wood 

AFHRL/FTXR 

Howard  L.  Parris 

AFHRL/XP 

Stanley  Brumaghim 

Boeing-Wichita 

John  B.  Dempster 

Boeing-Wichita 

James  J.  Regan 

Navy  Pers  R&D  Center 

Joseph  A.  Puig 

Naval  Tmg  Equip  Ctr 

W.  G.  Matheny 

Life  Sciences,  Inc. 

Bill  Albery 

AFHRL/ASM 

F.  Thomas  Eggemeier 

AFHRL/ASR 

R.  N.  Burrell 

McDonnell  Douglas 

Col  S.  E.  Shrum 

ATC/XPT 

Lt  Col  E.  A.  Crimp 

USAFTAWC/OLAH 

Capt  George  Buckland 

AFHRL/FTS 

Dave  Kase 

Rockwell/ B-1  Div 

George  Kitchel 

Vought  Corp 

Cdr  J.  A.  Cade 

CNET 

Roland  L.  Bowles 

NASA/ Langley 

Capt  John  M.  Garrity 

AFHRL/DOPF 

Lt  Col  H.  A.  Hornbarger 

ATC/XPQC 

John  D.  Smith 

Logicon,  Inc. 

King  Povermire 

Coast  Guard,  ATC 

Maj  Jeff  Koonce 

DFBL/USAF  Academy 

Capt  Jack  Thorpe 

afhruftt 

Wayne  L.  Waag 

AFHRL/FTT 

Capt  Gene  Englund 

Hq  SAC 

Capt  Oak  DeBerg 

ASD/ENECC 
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I Grade/Name 

[ Capt  Barry  McFarland 

I Capt  Terry  Balven 

I Col  Rondel  E.  Minter 

Anchard  F.  Zeller 
Capt  David  L.  Miller 
John  B.  Shinn 
Capt  Steven  Rust 
Thomas  W.  Sellers 
Alfred  R.  Fregly 
Stanley  N.  Roscoe 

Donald  W.  Davidson 

i Conrad  L.  Kraft 

Bill  Harris 
W.  J.  Allsopp 

W.  A.  Upman 
E.  W.  Cairns 
R.  H.  Mathews 
Gary  McCulloch 
Grant  Beutler 
W.  E.  “Bud”  Nylen 
R.  E.  (Bob)  Coward 
R.  N.  (Bob)  Hale 
W.  D.  (Bill)  Hayden 
H.  T.  Bruel 
T.  L.  Keller 
R.  G.  Palmer 
J.  N.  Puglisi 
Cdr  Hugh  Halpin 
Cdr  Jim  Murray 

R.  F.  Browning 
G.  A.  Wynn 

Warren  E.  Richeson 
Capt  Dwight  Kelly 
James  E.  Brown 

Maj  Robert  E.  MacArgel 

Capt  Jerry  Stecklein 
Edward  A.  Stark 

Dom  Gibino 
Don  R.  Gum 

Barry  Leshowitz 

Col  Carl  G.  Baily 

Lt  Col  J.  H.  Snelling 

Frank  Schufletowski 

Pete  Gadwa 

I Bob  Lawson 


Organization 

ASD/ENECC 
ASD/SD24 
AF/IGD  (SEF) 
AFISE/SEL 
AFISC/SES 
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SIMULATOR  TRAINING  RESEARCH  PLANNING  MEETING 


21  AprU  0800  Registration,  Coffee  at  Building  558,  Lobby 

0830  Session  1.  Welcome  — Col  J.  D.  Boren,  AFHRL/FT 

Objectives  — Col  C.  H.  Markwood,  ASD/SM24 
Orientation  — Dr.  E.  E.  Eddowes,  AFHRL/FT 
Bldg  571,  Room  1 1 

0930  Session  2.  Study  Group  Meetings 

Group  1 . Simulator  Capabilities  and  Characteristics, 

Bldg  571,  Room  1 1 

Group  2.  Simulator  Requirements  for  Hying  Training, 
Bldg  571,  Room  5 

Group  3.  Measurement  of  Simulator  Training  Effective- 
ness, Bldg  S-561 

Group  4.  Management  of  Procurement  and  Utilization  of 
Simulators  for  Flying  Training,  Bldg  558, 
Conference  Room 

1 130  Lunch,  Williams  AFB  Officers’  Qub 
1 300  Study  Group  Meetings 

1630  Adjourn 

1 700  No-Host  Cocktail  Hour,  Williams  AFB  Officers’  Club 
1900  Tours  of  AFHRL  Facility 

22  April  0800  Coffee  at  Building  558,  Lobby 

0830  Report  of  Study  Group  Findings,  Bldg  571 , Room  1 1 
Group  1 — 0830 
Group  2 —0915 
Group  3 - 1000 
Group  4 - 1045 

1 130  Lunch,  Williams  AFB  Officers’  Club 

1 300  Discussion  of  Study  Group  Reports 

1430  Summary  of  Results 

1 500  Adjourn 
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Study  Group  1,  Building  571,  Room  1 1 
SIMULATOR  CAPABILITIES  AND  CHARACTERISTICS 


Moderator;  Mr.  Gum,  AFHRL/AS 

Mr.  Albery,  AFHRL/AS 

Mr.  Allsopt,  Boeing  Aircraft 

Mr.  Barger,  Singer  SPD 

Mr.  Beardsley,  GE  Daytona  Beach 

Capt  Burke,  TAC/DOXS 

Mr.  Burrell,  McDonnell  Douglas 

Mr.  Cairns,  Hydra  Systems 

Mr.  Case,  B-1  Division 

Mr.  Davidson,  Goodyear  Aerospace 

Mr.  Dusterberry,  NASA/ AMES 

Capt  Englund,  SAC/XPHV 

Mr.  Hale,  Vought  Corp 

Mr.  Harris,  NTEC 

Mr.  Hayden,  Vought  Corp 

Mr.  Heinle,  Logicon 


Lt  Col  Homberger,  ATC 

Mr.  Keller,  Grumman  Aerospace 

Mr.  Kiron,  Singer  SPD 

Mr.  Kitchel,  Vought  Corp 

Mr.  Mathews,  McDonnell  Douglas 

Col  Minter,  USAF  Safety  Center 

Mr.  New,  Singer  SPD 

Mr.  Nylen,  United  Airlines 

Mr.  O’Connor,  Singer  SPD 

Mr.  Palmer,  NTEC 

Ms.  Puglisi,  NTEC 

Mr.  Sellers,  Austin  Electronics 

Dr.  Shiner,  GE  Daytona  Beach 

Mr.  Smith,  Logicon 


Study  Group  2,  Building  571,  Room  5 
SIMULATOR  REQUIREMENTS  FOR  AIR  FORCE  FLYING  TRAINING 


Moderators;  Dr.  Waag  and  Dr.  Pohlmann,  AFHRL/FT 


Col  Baker,  USAF  Safety  Center 
Capt  DePerg.  ASD/ENEC 
Lt  Col  DeMuth,  SAC/XPHV 
Mr.  Jellison,  Decision  Sciences 
Maj  MacArgel,  TAWC 
Col  Markwood,  ASD/SD24 


Mr.  McCullock,  United  Airlines 
Mr.  Morton,  CNETS 
Col  Shrum,  ATC/XPT 
Lt  Col  Snelling,  CNATRA 
Capt  Stecklein,  AF/XOOFB 
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Study  Group  3,  Building  S-5S61 

MEASUREMENT  OF  SIMULATOR  TRAINING  EFFECTIVENESS 


Moderators:  Capt  Thorpe,  Capt  Fuller,  AFHRL/FT.  and  Capt  Rust,  TAWC 

Mr.  Beutler,  United  Airlines 
Dr.  Bowles,  NASA/ Langley 
Mr.  Browning,  NTEC 
Dr.  Eggemeier,  AFHRL/AS 
Dr.  Freitag,  CNETS 

Dr.  Hopkins,  Aviation  Research  Laboratory,  University  of  Olinois 

Dr.  Houston,  American  Airlines 

Maj  Koonce,  AFA/DFBL 

Dr  Matheny,  life  Sciences 

Capt  McFarland,  ASD/ENEC 

Capt  Miller,  USAF  Safety  Center 

Dr.  Payne,  Decision  Sciences 

Mr.  Puig,  NTEC 

Dr.  Shufletowski,  CNATRA 

Dr.  Stark,  Singer  SPD 


Study  Group  4,  Building  558,  Conference  Room 

MANAGEMENT  OF  PROCUREMENT  AND  UTILIZATION  OF 
SIMULATORS  FOR  FLYING  TRAINING 


Moderators:  Dr  Eddowes  and  Dr  King,  AFHRL/FT 

CDRCade,CNET 

Dr.  Fregley,  AFOSR 

Dr.  Fogel,  Decision  Sciences 

Dr.  Orlansky,  Institute  for  Defense  Analysis 

Dr.  Parris,  AFHRL/XP 

Dr.  Regan,  NPRDC 

Dr.  Roscoe,  Hughes  Aircraft 

Capt  Thomas,  CNO/OP5% 

CDR  Windsor,  CNATRA 
Dr.  Zeller,  USAF  Safety  Center 
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RESEARCH  VARUBLES 

The  research  program  envisioned  to  investigate  the  training  effectiveness  of  flight  simulator  motion  and 
visual  systems  will  be  focused  on  the  following  independent  and  dependent  variables: 

A.  Independent  Variables: 

1 . Simulator  Characteristics 

a.  Motion  Generation  Systems; 

1.  Platform  motion 

2.  G-seat 

3.  G-suit 

b.  Visual  Display  Systems 

1.  Model  board 

2.  Caligraphic 

3.  Computer  generated  imagery 

2.  Pilot  Rying  Experience  Level 

a.  Novice 

b.  Low  (UPT  graduate) 

c.  Medium 

d.  High  (Airlines  Captain) 

3.  Type  Aircraft  Simulated 

a.  Performance  Characteristics 

1.  Low 

2.  Medium 

3.  High 

b.  Size,  Weight  and  Complexity 

1.  Low 

2.  Medium 

3.  High 

4.  Training  Maneuvers 

a.  Maneuvers  expected  to  require  motion  simulation  to  produce 
positive  transfer  to  the  aircraft. 

b.  Maneuvers  not  expected  to  require  motion  simulation  to  produce 
positive  transfer  to  the  aircraft. 

B.  Dependent  Variables; 

1 . Training  effectiveness,  as  shown  by  transfer  of  training  to  the 
aircraft. 

2.  Pilot  performance  in  the  simulator. 

3.  Pilot  acceptance  ratings  of  the  simulator  motion  and  visual 
conditions  tested. 

4.  Pilot  discrimination  of  the  presence  or  absence  of  simulator  motion. 
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Study  Group  1.  SIMULATOR  CAPABILITIES  AND  CHARACTERISTICS 
Research  Discussion  Questions 


1.  How  many  different  types  of  motion  systems  should  be  tested  in  terms  of;  degrees  of  freedom  of 
platform  motion,  motion  drive  philosophy,  platform  motion  mechanization,  G-suit  and  G-seat,  or  others? 
The  issue  here  is  what  array  of  motion  generation  alternatives  is  so  provocative,  inexpensive,  reliable, 
acceptable,  etc,  as  to  require  full  research  evaluation? 

2.  How  many  different  visual  simulation  system  features  must  be  tested?  For  example,  field-of-view, 
black-and-white  vs.  color  CIG,  relative  enrichment  of  scene  presented,  modifiability  of  scene  content,  initial 
costs  vs.  refinement  and  update  costs,  etc. 

3.  Is  there  any  feasible  means  available  to  evaluate  the  likely  impact  of  state-of-the-art  advances  in  motion 
simulation  systems  using  currently  available  devices?  That  is,  are  convincing  extrapolations  likely  to  result 
from  this  research  program,  or  must  we  revalidate  all  the  evaluations  next  month  or  next  year  if  a “new 
improvement”  becomes  available? 

4.  What  biasing  factors  are  there  known  to  engineering  specialists  which  should  be  signaled  to 
decision-makers  to  aid  them  in  interpreting  research  outcomes?  That  is,  is  there  a convincing  way  to  portray 
the  current  consensus  on  the  inherent  strengths  and  limitations  of  available  simulator  motion  and  visual 
systems? 

5.  What  criteria  may  be  used  to  determine  if  research  or  training  simulators,  such  as  may  be  available  for 
use  in  this  research  program,  and  can  furnish  a set  of  pilot  perfomiance  data  which  decision-makers  can  use 
in  determining  what  simulation  motion/visual  systems  to  buy? 

6.  Are  there  devices  that  may  be  added  to  simulators  for  the  purpose  of  collecting  quantitative  pilot 
performance  measurements,  or  must  this  feature  be  provided  through  modification  of  the  simulator?  Wliat 
costs  are  involved  in  either  of  these  possibilities?  What  other  alternatives  are  available? 

7.  Others. . . . 


Study  Group  2.  SIMULATOR  REQUIREMENTS  FOR  FLYING  TRAINING 
Research  Discussion  Questions 


1.  How  will  simulators  be  used  in  the  flying  training  programs  of  the  various  major  commands 
(MAJCOMs)? 

2.  Are  there  key  missions/ maneuvers  that  are  to  be  trained  using  simulators  equipped  with  motion  and 
visual  systems  which  must  be  a part  of  any  evaluation  of  these  simulator  capabilities? 

3.  What  flying  experience  levels  are  characteristicaUy  found  among  various  MAJCOM  pilots  who  use 
simulators  in  their  flying  training? 

4.  Are  there  known  biases  among  the  pilots  who  are  potential  users  of  simulators  in  favor  of  or  against 
simulators  equipped  with  motion  and/or  visual  systems?  What  are  the  characteristics  of  such  pilot  biases  as 
may  exist? 

5.  Will  the  same  simulators  be  used  for  cockpit  procedures,  emergency  procedures,  instrument  procedures 
and  full  mission  training?  If  not,  what  other  training  devices  may  be  expected  to  be  used  in  MAJCOM  flying 
training  programs? 
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6.  What  types  of  data  on  simulator  effects  on  pilot  performance  are  likely  to  be  convincing  to  MAJCOM 
training  managers;  training  transfer  effectiveness,  performance  in  the  simulator,  pilot  acceptance,  pilot 
discrimination  of  the  presence  or  absence  of  motion,  others? 

7.  Others. . . . 


Study  Group  3.  MEASUREMENT  OF  SIMULATOR  TRAINING  EFFECTIVENESS 
Research  Discussion  Questions 


1 . What  are  the  relative  merits  and  shortcomings  of  the  various  experimental  designs  which  can  be  used  to 
study  the  training  effectiveness  of  motion? 

--  Training  (transfer)  effectiveness 

- In-simulator  performance  comparisons 

- Pilot  discrimination  of  the  presence  or  absence  of  simulator  motion 

- Pilot  acceptance  or  attitude 

2.  What  techniques  have  been  used  in  the  past  for  testing  the  effects  of  simulator  characteristics  on  pilot 
transfer  to  the  aircraft? 

3.  What  are  tlie  practical  strengths  and  weaknesses  of  dependent  variables  used  in  these  designs? 

" Pilot  ratings  in  the  aircraft 

- Instrumented  aircraft  measures 

- Objective  performance  measures  in  the  simulator 

- Pilot  opinion  questionnaire  | 

4.  What  current  performance  measurement  systems  and  approaches,  including  pilot  rating  schemes, 
structured  rating  scales  and  rater  training  sessions  are  available  for  measuring  simulator  training 
effectiveness? 

5.  What  types  of  performance  differences  are  we  looking  for  and  what  should  their  magnitude  and 
duration  be? 

- How  large  should  a “significant”  difference  be  to  be  meaningful? 

- Are  we  mostly  concerned  with  short-term  effects,  i.e.,  those  which  can  be  observed  over  the  first 
few  flights,  or  are  longitudinal  effects  important,  and  if  they  are,  how  do  we  control  all  the  possible 
confounding  factors  in  a pilot  training  program  so  that  they  can  be  studied  with  some  degree  of 
orthogonality  (or  believability)? 

6.  How  will  the  problem  of  generalization  be  resolved?  What  rationale  can  be  offered  to  relate  the  results 
of  small  n studies  using  one  specific  simulator  and  aircraft  type  combination  to  the  data  generated  in 
another  simulator-aircraft  combination  or  to  another  simulator-aircraft  combination  where  no  studies  have 
been  conducted? 

~ Are  different  rationales  needed  to  relate  such  differences  in  pilot  experience  levels,  maneuver  types, 
simulator  characteristics,  (motion,  G-seat.  visual  system)  and  aircraft  type  as  arc  studied? 

7.  How  do  we  avoid  proving  the  null  hypotheses? 

- Are  there  procedures  that  can  be  used  to  prevent  experimental  outcomes  which  show  no 
differences? 

- Are  decisions  based  on  the  cost  of  other  features  or  devices  which  generate  no  measurable 
performance  effects  supportable? 

8.  What  is  a reasonable  heirarchy  of  studies  to  provide  the  most  potent  information  earliest? 

9.  Is  there  a best  way  to  study  new  configurations  proposed  by  contractors  (for  example,  in  the  search  for 
“good”  motion)? 

10.  Others. . . . 
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Study  Group  4.  MANAGEMENT  OF  PROCUREMENT  AND  UTILIZATION  OF 
SIMULATORS  FOR  FLYING  TRAINING 


Research  Discussion  Questions 


1.  What  attitudes  or  positions  are  found  frequently  among  Air  Force/Navy  training  managers  on  the 
absolute  or  relative  value  of  simulator  motion  and  visual  systems? 

2.  Is  there  any  tendency  among  training  managers  to  consider  training  effectiveness  factors  beyond  pilot 
acceptance  and  cost?  If  so,  what  are  the  most  salient  of  the  other  factors? 

3.  What  is  the  likely  response  of  flying  training  managers  to  a proposal  that  any  study  of  simulator  training 
effectiveness  should  include  the  optimization  of  the  integration  of  ground  training,  simulator  and  flight 
training?  Can  you  detect  any  strong  feelings  that  integrated  ground  and  flying  training  is  a m^or  goal  of 
future  flying  training  programs? 

4.  There  has  been  much  attention  focused  on  reducing  flying  time.  Has  there  been  much  attention  focused 
on  how  to  best  use  what  flying  time  may  be  available?  For  example,  can  enough  flying  hours  be  gained  by 
giving  instrument  training  and  flight  checks  in  the  simulator  to  permit  redistribution  of  flying  hours  to 
missions/maneuvers  less  easy  or  less  expensive  to  simulate  on  the  ground? 

5.  In  managing  flying  training,  is  there  any  possibility  that  a competitive  cost  reduction  approach  may  be 
the  best  route  to  success?  For  example,  establish  two  or  more  competitive  teams  with  the  objective  of 
generating  an  optimum  mix  of  ground  and  flight  training  and  then  evaluating  the  effectiveness  of  the 
resulting  programs  in  terms  of  yearly  dollar  costs  required  to  maintain  force  readiness,  or  in  terms  of 
weapons  delivery  data  or  flight  hours  saved  to  attain  some  agreed-to  performance  criterion  (such  as  an  FAA 
check).  Might  incentives  be  used  such  as  allocating  all  cost  savings  to  provide  additional  aircraft  training 
time? 

6.  What  is  the  likelihood  of  acceptance  by  training  managers  of  radical  changes  in  flying  training  program 
structure?  For  example,  giving  flight  hour  credit  for  simulator  time  or  cash  incentives  to  pilots  who  qualify 
in  the  least  expensive  manner  for  their  mission-ready  status? 

I 7.  Is  there  any  conviction  among  training  managers  regarding  the  utility  or  desirability  of  accomplishing 

part  of  the  flying  training  requirements  in  a simplified  or  low-cost  aircraft  in  addition  to  the  mission 
j aircraft? 

I 8.  Others.  . . . 
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